1. Field of the Invention
The invention relates to simple input devices for computers, well suited for use with 3-D graphically intensive activities, and operating by optically sensing object or human positions and/or orientations. The invention in many preferred embodiments uses real time stereo photogrammetry using single or multiple TV cameras whose output is analyzed and used as input to a personal computer.
2. Description of Related Art
The closest known references to the stereo photogrammetric imaging of datum's employed by several preferred embodiments of the invention are thought to exist in the fields of flight simulation, robotics, animation and biomechanical studies. Some early prior art references in these fields are: Pugh U.S. Pat. No. 4,631,676; Birk U.S. Pat. No. 4,416,924; Pinckney U.S. Pat. No. 4,219,847; U.S. Pat. No. 4,672,564 by Egli et al, filed Nov. 15, 1984; Pryor U.S. Pat. No. 5,506,682, robot vision using targets; Pryor, Method for Automatically Handling, Assembling & Working on Objects U.S. Pat. No. 4,654,949; and Pryor, U.S. Pat. No. 5,148,591, Vision target based assembly.
In what is called “virtual reality”, a number of other devices have appeared for human instruction to a computer. Examples are head trackers, magnetic pickups on the human and the like, which have their counterpart in the invention herein.
References from this field having similar goals to some aspects of the invention herein are: U.S. Pat. No. 5,297,061 by Dementhon et al.; U.S. Pat. No. 5,388,059 also by Dementhon, et al.; U.S. Pat. No. 5,168,531: Real-time recognition of pointing information from video, by Sigel; U.S. Pat. No. 5,617,312: Computer system that enters control information by means of video camera by Iura et al., filed Nov. 18, 1994; U.S. Pat. No. 5,616,078: Motion-controlled video entertainment system, by Oh; Ketsu; U.S. Pat. No. 5,594,469: Hand gesture machine control system, by Feeman, et al.; U.S. Pat. No. 5,454,043: Dynamic and static hand gesture recognition through low-level image analysis by Freeman; U.S. Pat. No. 5,581,276: 3D human interface apparatus using motion recognition based on dynamic image processing, by Cipolla et al.; U.S. Pat. No. 4,843,568: Real time perception of and response to the actions of an unencumbered participant/user by Krueger, et al.
Iura and Sigel disclose means for using a video camera to look at an operator's body or finger and input control information to a computer. Their disclosure is generally limited to two dimensional inputs in an XY plane, such as would be traveled by a mouse used conventionally.
Dementhion discloses the use objects equipped with 4 LEDs detected with a single video camera to provide a 6 degree of freedom solution of object position and orientation. He downplays the use of retroreflector targets for this task.
Cipolla et al discusses processing and recognition of movement sequence gesture inputs detected with a single video camera whereby objects or parts of humans equipped with four reflective targets or LEDs are moved through space, and a sequence of images of the objects taken and processed. The targets can be colored to aid discrimination.
Pryor, one of the inventors, in several previous applications has described single and dual (stereo) camera systems utilizing natural features of objects or special targets including retroreflectors for determination of position and orientation of objects in real time suitable for computer input, in up to 6 degrees of freedom.
Pinckney has described a single camera method for using and detecting 4 reflective targets to determine position and orientation of an object in 6 degrees of freedom. A paper by Dr. H. F. L. Pinckney entitled Theory and Development of an on line 30 Hz video photogrammetry system for real-time 3 dimensional control presented at the Symposium of Commission V Photogrammetry for Industry, Stockholm, August 1978, together with many of the references referred to therein gives many of the underlying equations of solution of photogrammetry particularly with a single camera. Another reference relating to use of two or more cameras, is Development of Stereo Vision for Industrial Inspection, Dr. S. F. El-Hakim, Proceedings of the Instrument Society of America (ISA) Symposium, Calgary Alta, Apr. 3-5, 1989. This paper too has several useful references to the photogrammetry art.
Generally speaking, while several prior art references have provided pieces of the puzzle, none has disclosed a workable system capable of widespread use, the variety and scope of embodiments herein, nor the breath and novelty of applications made possible with electro-optical determination of object position and/or orientation.
In this invention, many embodiments may operate with natural features, colored targets, self-illuminated targets such as LEDs, or with retroreflective targets. Generally the latter two give the best results from the point of view of speed and reliability of detection—of major importance to widespread dissemination of the technology.
However, of these two, only the retroreflector is both low cost, and totally unobtrusive to the user. Despite certain problems using same, it is the preferred type of target for general use, at least for detection in more than 3 degrees of freedom. Even in only two degrees, where standard “blob” type image processing might reasonably be used to find ones finger for example, (see U.S. Pat. No. 5,168,531 by Sigel), use of simple glass bead based, or molded plastic corner cube based retroreflectors allows much higher frequency response (e.g. 30 Hz, 60 Hz, or even higher detection rates) from the multiple incidence angles needed in normal environments, also with lower cost computers under a wider variety of conditions—and is more reliable as well. (at least with today's PC processing power).